implement season data importer - #6
Conversation
`TODO: maybe save memory and delete DataFrame?`
didnt think far enough...
|
Thanks for this, it's a good start :) Currently this would lead to scraping the wiki for every game processed and filesystem caching isn't a practical way to mitigate this since the processing is serverless. I do like using an automated source for the data rather than being a manual process, but to make it work I think we would want to do a scheduled scrape to overtrack's own infrastructure. This would also remove Downloader/Parser: Loader: The parser would then run on a schedule and upload to S3 |
Implemented dynamic SeasonData importer utilizing https://apexlegends.fandom.com/wiki/Season
Also providing
has_duosandhas_arenasbool for game modes.TODO: maybe save memory and delete DataFrame?TODO: maybe implement Caching for website data